默认推荐的文本分割器是 RecursiveCharacterTextSplitter。该文本分割器接受一个字符列表。它尝试根据第一个字符进行分割来创建块，但如果任何块太大，则继续移动到下一个字符，依此类推。默认情况下，它尝试进行分割的字符是 `["\n\n", "\n", " ", ""]`

除了控制可以进行分割的字符之外，您还可以控制一些其他事项：

- `length_function`：计算块长度的方法。默认只计算字符数，但通常在此处传递一个令牌计数器。- `chunk_size`：块的最大大小（由长度函数测量）。- `chunk_overlap`：块之间的最大重叠。保持一些连续性之间可能有一些重叠（例如使用滑动窗口）。- `add_start_index`：是否在元数据中包含每个块在原始文档中的起始位置。

```python
This is a long document we can split up.
with open('../../state_of_the_union.txt') as f:
    state_of_the_union = f.read()
```


```python
from langchain.text_splitter import RecursiveCharacterTextSplitter
```


```python
text_splitter = RecursiveCharacterTextSplitter(
    # Set a really small chunk size, just to show.
    chunk_size = 100,
    chunk_overlap  = 20,
    length_function = len,
    add_start_index = True,
)
```


```python
texts = text_splitter.create_documents([state_of_the_union])
print(texts[0])
print(texts[1])
```

<CodeOutputBlock lang="python">

```
    page_content='Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and' metadata={'start_index': 0}
    page_content='of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.' metadata={'start_index': 82}
```

</CodeOutputBlock>
